[% setvar title Normalisation and C<unicode::exact> %]
<div id="archive-notice">
    <h3>This file is part of the Perl 6 Archive</h3>
    <p>To see what is currently happening visit <a href="http://www.perl6.org/">http://www.perl6.org/</a></p>
</div>
<div class='pod'>
<a name='TITLE'></a><h1>TITLE</h1>
<p>Normalisation and <code>unicode::exact</code></p>
<a name='VERSION'></a><h1>VERSION</h1>
<pre>  Maintainer: Simon Cozens &lt;<a href='mailto:simon@brecon.co.uk'>simon@brecon.co.uk</a>&gt;
  Date: 25 Sep 2000
  Mailing List: <a href='mailto:perl6-internals@perl.org'>perl6-internals@perl.org</a>
  Number: 295
  Version: 1
  Status: Developing</pre>
<a name='ABSTRACT'></a><h1>ABSTRACT</h1>
<p>Perl 6 should support Unicode normalisation; this is going to make
comparing strings confusing.</p>
<a name='DESCRIPTION'></a><h1>DESCRIPTION</h1>
<p>First, what's normalisation? Unicode gives the user a lot of flexibility
over how data is represented. For instance, there are two ways of
representing &eacute;; (that's an e with an acute accent) first,
there's <code>U+00E9</code>, (Also handily named LATIN SMALL LETTER E WITH ACUTE)
and there's secondly the two characters <code>U+0065 U+0301</code>. (That's a
an acute accent which combines with an ordinary latin letter 'e')</p>
<p>Normalisation is the process of turning all data in the first type of
representation to the second type. Well, strictly speaking, this is
&quot;decomposition&quot;, but the purpose of it is that we can now compare things
for their meaning, and not solely for their representation, and doing it
for that purpose is normalisation.</p>
<p>Perl 6 should support normalisation. But this creates a problem. Should
the <code>eq</code> operator compare representations or meanings? After plying the
perl5-porters with large quantities of alcohol at YAPC::Europe, the
consensus was that it should compare meanings. Good. Perl's always been
about handling text for meaning. But then how can we tell whether two
strings are really equal in terms of their representation?</p>
<a name='IMPLEMENTATION'></a><h1>IMPLEMENTATION</h1>
<p>The current <code>use bytes</code> pragma (which is the subject of another of my
Unicode RFCs) will allow comparison in terms of representation; there's
also a problem of optimisation.</p>
<p>If we keep the original data, we need to perform decomposition every
time we do any kind of string comparison, and I don't relish the
prospect of <code>cmp</code> becoming really slow. Of course, you could store a
decomposed PV inside the SV as well, but that's big and heavy; the only
sensible, non-destructive optimisation would be to have some kind of
<code>IsNormalised</code> flag in the SV which tells us not to bother decomposing
this string, since it's already in a normalised representation.</p>
<p>I propose that by default, Perl 6 is allowed to chew up your data and
decompose it. If the exact representation is important to the user, the
pragma <code>unicode::exact</code> should be turned on; inside of the scope of
<code>unicode::exact</code>, no normalisation is performed, and <code>cmp</code> and friends
perform normalisation on a temporary copy of the string so as to be
non-destructive to the original data. For instance:</p>
<pre>    $x = chr(0x00E9); # LATIN SMALL LETTER E WITH ACUTE
    ... if ($x cmp $y);
    # $x is *actually* chr(0x0065).chr(0x0301) now
    
    {
        use unicode::exact;
        $x = chr(0x00E9);
        ... if ($x cmp $y);
        # $x is compared as if it were chr(0x0065).chr(0x0301),
        # but it retains its old value of chr(0x00E9)
    }</pre>
<p>Outside of <code>unicode::exact</code>, whether the normalisation is done lazily
(necessitating an <code>IsNormalised</code> flag) or when the data is stored is
not specified by this RFC; it works fine both ways. I'd personally say
it should be done on lazily.</p>
<a name='REFERENCES'></a><h1>REFERENCES</h1>
<p>The Unicode FAQ; <a href='http://www.unicode.org/unicode/faq' target='_blank'>www.unicode.org</a></p>
<p>RFC 300: <code>use unicode::representation</code></p>
<p>RFC 312: Unicode Combinatorix</p>
<p>RFC ??:When UTF8 Leaks Out</p>
</div>
